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Parent Involvement Data: How to Measure and 
Improve Representativeness for Indicator B8 


Indicator 8: Percent of parents with a child receiving special education services who 
report that schools facilitated parent involvement as a means of improving services 
and results for children with disabilities (20 U.S.C. §1416(a)(3)(A)). 


Representativeness is a critical component of data quality. For Indicator 8, representativeness refers to the 
extent to which the demographics of the children and youth whose parents provided data are representative 
of the demographics of al/ children and youth receiving special education services in the state. If a state’s 
Indicator 8 data are not representative, the data most likely will not accurately reflect the experiences of 
parents in the state. This inaccuracy occurs when the opinions or factors for certain groups of parents who 
complete the Indicator 8 survey differ in meaningful ways from those who do not. Strategies that can help 
states collect representative data and evaluate and improve the representativeness of their data before, 
during, and after data collection follow. 


Collecting Data That Are Representative 


The Office of Special Education Programs (OSEP) requires states to analyze the extent to which the 
demographics of the children and youth of parents who responded to a survey are representative of the 
demographics of children and youth receiving special education services. This effort begins with planning and 
conducting data collection. Beginning with the FFY 2021 SPP/APR submission due in 2023, states must 
consider race and ethnicity and at least one other category from the examples below or another category 
that they selected with stakeholder input: 


e —_ race/ethnicity (required); 

e gender; 

e student age; 

e disability category; 

e geographic location in the state; and 


e other category selected with stakeholder input. 


Using a Census or Sample 


States collect information for Indicator 8 using a census (asking all parents of children and youth receiving 
services to participate) or a sample (asking only selected parents to participate). Using a sample saves time 
and money and, when done correctly, can help improve the quality and accuracy of the data by leaving more 
time and resources for monitoring the data as they arrive and following up with parents as necessary. 
However, selecting a representative sample can be complicated. (See Sampling for more information.) 
Whether states choose a census or a representative sample for Indicator 8 data collection, they will need to 
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use strategies that encourage participation and data collection methods that ensure they have critical 


information on representativeness. 


Encouraging Participation 


Survey design and data collection methods can affect whether parents respond fully, or at all. States should 
design their survey to make it easy for parents to respond. High response rates make it more likely that the 
sample will be representative. (See Response Rates for more information.) States can encourage parents to 


respond fully and accurately by making sure 


e questions are engaging and easy to understand; 


e options are clear and categories don’t overlap; 


e surveys are easily accessible, considering availability online and paper options, use of multiple 
languages, Section 508 compliance, and multiple methods for delivery and notifications; 


e surveys include a clear explanation of purpose and how participation will benefit parents, students, and 


schools; and 


e they have a plan to follow-up with those who do not respond initially. 


Gathering the Data 


States need to plan carefully and gather specific information so 
they can evaluate their results and fulfill reporting requirements 
of the State Performance Plan/Annual Performance Report 
(SPP/APR). Several strategies that can help states gather the 
most complete data follow. 


e States should gather data confidentially but not 
anonymously. Anonymous data cannot be traced back to 
the respondents while confidential data could be traced 
back; however, to protect the respondents, individual 
responses are not revealed outside the data collection 
team. In other words, states should track survey recipients 
and respondents but keep that information secure, keeping 
the names of respondents and their individual responses 
confidential. 


e States should make sure they have access, if possible, to all 
the demographic information they need before they start. If 
they cannot access needed information through other 
systems, states will need to ask questions in the survey to 
gain the information. Information states might want to 
consider, beyond what OSEP suggests, includes length of 
time a child received services, family income, and primary 
language. 


e States should ensure they have accurate contact 
information for parents (e.g., asking parents to verify 
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Problems with 
anonymous data 


It prevents states from 
tracking responses. They will 
not know whether the parents 
they asked to participate in 
the survey did not respond or 
accidentally responded more 
than once. 


It means that states have to 
ask all demographic questions 
in the survey, increasing the 
survey’s length and burden on 
parents. 


It increases the likelihood of 
missing data if parents skip 
demographic questions. 


It forces states to send out 
reminders to all parents, 
wasting resources and 
potentially annoying those 
who already responded. 


IDC 


Parent Involvement Data: How to Measure and Improve Representativeness 


contact information at individualized education program [IEP] meetings, verifying with districts that 
contact information is complete and accurate). 


e States should implement a system to track surveys. Many online survey services automatically combine 
survey tracking and response monitoring in a single database, allowing states to monitor response rates 
in real time. Online surveys provide the additional benefit of not requiring data entry. 


e States should include a unique identifier (ID) on each survey so they can track responses and match 
surveys to existing student demographic information. Some online services do this automatically, but 
states would need to add the ID number manually to paper surveys. With a unique identifier, states can 
link survey data with existing demographic data in other databases. 


e States should keep data secure. They should separate files that contain identifiers from those with 
survey responses, encrypt personally identifiable information, and limit access to the electronic and 
hardcopy data. 


Evaluating the Representativeness of the Data 


There are two ways to think about representativeness of data, both of which relate to whether or not the 
data come from respondents who are representative of the actual population and subpopulations of interest. 
One is whether the demographics of the respondents correspond to the demographics of the population on 
which the indicator focuses—for Indicator 8 that is the entire population of students with disabilities in the 
state. The other is whether the experiences of those who responded are similar enough to those who did not 
respond so that states can assume that their responses would have been similar. Keep in mind that although 
the respondents for Indicator B8 are parents, the issue of representativeness focuses on the demographics of 
their children, not the parents themselves. 


Representativeness of the Respondents 


Evaluating the representativeness of respondents is similar to asking, “Were parents of children and youth 
with certain characteristics more or less likely to respond to the survey than others?” States should compare 
results in the categories OSEP suggests (i.e., race/ethnicity, student gender, student age, disability category, 
and geographical location) and any additional categories they might wish to explore, such as income level, 
length of time child received services, or primary language. For each demographic category, states will need 
to compare the response percentages in the category to the statewide percentage of all students with 
disabilities who are in that category. 
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Consider Figure 1, with hypothetical data from Figure 1. 


State A. Children and youth with autism are 
overrepresented in the data because the 
percentage of respondents in that category is 
much higher than the percentage of children 
and youth in that category statewide. On the 
other hand, children and youth with speech or 
language impairments are underrepresented 
because their proportion of respondents is 
smaller than the proportion of students in that 
category statewide. Only in the category of 
developmental delay is the collected data 
proportional to the population of the target 
group. Because these data do not fairly 
represent some groups, the overall data 
cannot be said to be a good representation of 
all parents of children and youth with 
disabilities in the state. 
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How much of a difference between respondents and the target group is still considered representative? 
There is no one “threshold” for determining whether data are representative, but OSEP requires that states 
describe the metric they used to determine representativeness. States commonly use either a threshold of 
plus or minus three percentage points or statistical significance as their threshold for representativeness. 
Two tools that states are using to determine if their data meet these thresholds, although not designed 


specifically for Indicator 8, are 


e The National Technical Assistance Center on Transition’s (NTACT) Response Calculator and Instructions, 
which calculates differences in percentage points and uses a threshold of +3 percentage points. This 
calculator may indicate that responses are not representative unless there are large numbers in all 


groups a State is reviewing. 


e The Early Childhood Technical Assistance Center’s (ECTA) Response Rate and Representativeness 
Calculator, which uses tests of statistically significant differences. This tool works well even with small 
group sizes, but it is only appropriate to use with a census that is relatively free from nonresponse bias 


or with simple random sampling. 


Representativeness of the Responses 


Regardless of whether response rates are high or respondents are representative, states must consider the 
likelihood of nonresponse bias. Nonresponse bias occurs when those who did not respond to the survey are 
different in meaningful ways from those who did respond. These differences might be revealed when states 
examine the representativeness of their target groups, but other cases of nonresponse bias can occur even 


when respondents appear to be representative of the overall population. For example, nonresponse bias 
could happen if parents who are less involved in their children’s education were less likely to respond than 
parents who are more involved. That would lead to bias in the survey results by underrepresenting the 
opinions and experiences of parents who are less involved. If that happened, decisions based on those survey 
results would be faulty, as they likely would not address the needs of all parents and children and youth. 
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States must analyze their response rate to identify potential nonresponse bias and take steps to reduce any 
identified bias and promote responses from a broad cross-section of parents of children and youth with 
disabilities. States should plan to conduct nonresponse follow-up to minimize the possibility of bias and take 
steps to check for nonresponse bias after they collect data. One way to investigate nonresponse bias is by 
examining the responses that came in from parents at the end of the data collection period, as a proxy for 
nonresponders, compared to responses that came in during the beginning and middle of the data collection 
period. This method does not add additional costs to the data collection and provides insight into whether or 
not results may be biased. 


A better, but more expensive, way to assess nonresponse bias is to assess directly if there are differences 
between responders and nonresponders. To do this, states will need to make extra efforts to obtain survey 
responses from a sample of parents who initially did not respond to see if their responses differ from ones 
the states already have. To collect data from those who originally did not respond, states should follow these 
steps: 


1. Decide ona strategy to encourage responses. States should consider options such as using a shorter 
questionnaire (perhaps just the questions they are using to calculate Indicator 8) or offering incentives 
like small payments or gift cards. 


2. Randomly select a group of 10 to20 percent of the parents who did not respond to the survey during the 
initial data collection window and reach out to them. 


3. Contact the parents (usually by phone) to gather responses. 


4. Document the responses. For instance, states might complete a paper survey for parents during a phone 
call or enter their responses into an online survey. 


Once states have responses from their group of initial nonresponders, they can compare the responses to the 
responses they received originally. In the comparison, states should examine the degree of difference 
between the two groups to determine if it is meaningful. States should consider that some difference in 
responses could be due to the collection method if they used phone calls for non-responders and another 
method for the original data collection. To determine whether there are meaningful differences, states could 
use tests of statistical significance between the two groups of responders. States should consult with a staff 
member or contractor with statistical expertise to assist with the analysis. States also can contact their IDC 
State Liaison for assistance. 


Monitoring and Addressing Issues With 
Representativeness 


Monitoring representativeness during data collection is helpful because it allows states to make changes to 
boost responses and representativeness before collection is complete. After data collection, the chance to 
improve representativeness ends, but states still can make changes to balance the current data and improve 
the next round of data collection. 


During Data Collection 


Regular checks for representativeness during data collection allow states to identify issues in a timely manner 
and quickly implement steps to help improve representativeness. States should consider checking at least 
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twice—when they have received responses from about one-third and two-thirds of the expected number of 
parents. Without these checks, states will lose the opportunity to improve representativeness in that round 
of data collection. 


If states identify representativeness issues during data collection, they will need to work to address those 
issues as quickly as possible while collection continues. The most effective way to improve 
representativeness during data collection is to make extra efforts to encourage participation from groups 
that are underresponding. These efforts could include 


° sending additional reminders using different modes of communication, such as social media, mail, or 
phone calls, to parents in the underresponding groups; 

e — enlisting support from Parent Centers or other key stakeholder groups to encourage parents from 
underresponding groups to respond; and 

e calling parents who started but did not complete the survey to encourage them to respond either in the 
original format (e.g., online or mail) or during the phone call. 


After Data Collection 


When data collection is complete, states should assess representativeness again. If they have been careful 
when selecting their sample (if they sampled) and have tracked responses, monitored representativeness, 
and followed up with nonresponders, there is a good chance that they will have relatively representative 
data. 


If, after all of their efforts, their data are still not representative, states can consider statistical adjustment to 
mimic representative data such as “weighting.” Weighting re-balances the data to reflect the target 
populations better. It is a procedure by which states count data from some groups more or less than data 
from other groups to compensate for a lack of representativeness in the data they originally gathered. 
Weighting is complicated, and states often need multiple weighting techniques, in which case the final 
weight would be the product of all weighting techniques. Because of this complexity, states should consult 
with staff or consultants with statistical training and specific expertise in weighting to use weighting 
techniques. 


A more straightforward, but perhaps less adequate, way to treat non-representative or potentially biased 
data is to be transparent in reporting the data. States can explain areas of under- or over-representation in 
their results and narrative and caution the reader to interpret the results accordingly. States also should 
describe the results in terms of parents who completed the survey (i.e., “survey responders”) rather 
than all parents of children and youth with disabilities, while keeping in mind that data may be 
representative for certain subgroups with high response rates but not for the total population. 


Planning for Next Time 


The next round of data collection offers the opportunity to improve the representativeness of the data. OSEP 
requires states that reported non-representative data to describe their strategies to improve 
representativeness the next year. To make their plans, states should consider issues that they encountered in 
their last data collection and how they might correct them. Were there methods or strategies that were more 
successful? If so, states should implement them earlier and with greater intensity in the next round. For 
example, if states find that certain groups are underrepresented in their survey responders, in the next year’s 
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data collection, they may opt to increase outreach to that group, add more survey delivery options such as 
social media, increase the number of follow-ups to parents, or take other steps designed to increase 
representativeness. If states conducted a census, they could consider sampling respondents instead to allow 
them to focus their efforts better among demographic groups. (See Sampling for more information.) 


For questions, more information about representativeness, or assistance regarding analyzing Indicator 8 data 
for representativeness, states can contact their IDC State Liaison. 


Sampling 


Sampling can save time and resources compared to attempting to survey all parents of children and youth 
with disabilities, and it can result in representative data when done well. However, sampling is complex. 
Therefore, states should consult with someone with expertise in sampling to develop and implement their 
sampling plan, beginning with selecting the appropriate sample size and the appropriate sampling 
procedures. For effective consultation, it is helpful to know some basics, so information on sampling methods 
and creating a sampling plan follow. 


Selected Types of Sampling Methods 


There are many different types of sampling methods but not all are appropriate for use with parent 
involvement data. A discussion of a few selected common types follows. 


Random Sampling 


This method refers to samples selected completely at random from everyone eligible to participate. Random 
sampling is often preferred to other sampling methods because it should, in theory, result in a representative 
sample, since every eligible parent has an equal chance of being included in the sample. Random sampling is 
not always representative, though, and sometimes a random sample will, by sheer bad luck, not be 
representative. 


Oversampling 


This method uses sampling to correct for a group that states expect to be underrepresented because of small 
numbers in the population or expected low response rate. Oversampling works by giving parents in that 
group a higher likelihood of being randomly selected than parents in other groups. This creates larger 
subsamples of parents from the target group to better represent that group’s views. This method can be 
effective in gathering enough data from the target group, but it requires special considerations when trying 
to report statewide data. If states choose this method, they would need to weight the data to account for the 
fact that different groups had different probabilities of being selected. For example, states might oversample 
Hispanic parents to ensure that the sample of Hispanic parents is large enough. However, if states combine 
data across ethnicity groups to get estimates for the entire state, they must apply weights to individual values 
to adjust for the different selection probabilities. 
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Probability Sampling 


For a probability sample, such as a stratified sample, states sort eligible participants into important categories 
(e.g., geographic location or race/ethnicity) and then sample from within those categories. Eligible 
participants don’t all have the same probability of being selected (as they would with random sampling), but 
all within a category have the same chance of being selected as others in that category, and all have some 
chance of being selected. This method ensures that there will be an appropriate number of parents from 
each category in the sample, as opposed to random sampling, which might over- or under- represent a 
category by chance. As with oversampling, states weight final results from each category by how likely 
parents in the category were to be selected to create unbiased estimates of the entire population. 


Purposeful Sampling 


Purposeful sampling is a form of non-probability sampling in which states rely on their own judgment when 
choosing the sample. It does not include any element of random selection. In addition, this limitation 
prevents states from being able to assess how representative the sample is. Therefore, this method is not 
appropriate to use for Indicator 8 data collection. 


Steps to Take When Sampling 


Here are specific steps states should take when sampling. 


1. Engage ateam member or a third-party contractor with specific training, expertise, or experience in 
sampling, since there are multiple technical details that states must consider. 


2. Develop a sampling plan that specifies the intended sample size and type of sampling method states 
plan to use to ensure the data are representative. 


3. Implement the sampling plan with fidelity. 


4. Review and, if needed, modify the sampling plan periodically, especially if the demographics across the 
state shifted since states determined their sampling plan. Changing demographics across or within 
certain districts could mean states need to examine their sampling strategy to ensure the 


representativeness of the sample. 


Response Rates 


OSEP requires states to report the number of parents to whom they distributed surveys and the number of 
parents who responded. States also must compare response rates from year to year for specific groups of 
parents and describe strategies they will implement to increase the response rate, particularly for those 
groups that are underrepresented. It is important that states monitor and report their response rate because 
low response rates can affect data quality. 


An overall high response rate alone does not guarantee representativeness. To know how representative 
their data are, states combine information about response rates with information about the demographic 
characteristics of their population of children and youth with disabilities. For example, if one demographic 
group in the state was less likely to respond than others were, the data will not represent all children and 
youth with disabilities in the state, even if the response rate overall was high (defined as 80% or higher by the 
Office of Management and Budget, 2006). Alternatively, if the characteristics of the state’s respondents 
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closely match those of all children and youth with disabilities in the state, the data may be representative 
even if the response rate was under 80 percent. 
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